Adaptive Data Partition Using Probability Distribution

Many computing problems benefit from dynamic partition of data into smaller chunks with better parallelism and locality. Previous partition methods either have high overhead or only apply to uniformly distributed data. This paper presents a new partition method in sorting scenario based on probability distribution, an idea first studied by Janus and Lamagna in early 1980’s on a mainframe computer. Our new method makes three contributions. The first is a rigorous sampling method that ensures accurate estimate of the probability distribution. The second is an efficient implementation on modern machines. Finally, it uses probability distribution in parallel sorting. Experiments show 1030% improvement in partition balance and 20-70% reduction in partition overhead, compared to approaches popular on current systems. When used in the classical problem of sorting, the method not only saves parallel sorting time by 33-50%, but also outperforms by up to 30% the fastest sequential sorting methods from the recent literature.